An item displayed on a computer screen may be considered a projection of an actual object; for instance, a displayed document can be thought of as a projection of an actual document, where the display shows one page, several pages, or part of a page at a time.
There are several conventional techniques for allowing a user to “scroll” a displayed image so that the user can see one portion and then move to the next (or some other) portion to see. There are also standard ways to “zoom” the display in and out, so that a user can either see all (e.g., an entire page or several pages at once), or only a portion (of one page), depending on what he wants to see and the size of his display screen. Almost all conventional methods and devices have separate controls for zooming and for scrolling.
Scrolling is not limited to documents inasmuch as pictures of objects, maps, etc. may all be displayed in a similar fashion. Thus, scrolling and zooming are the standard terms used for specific operations to look at different parts of a display regardless of what kind of image is displayed.
Hand-held devices are more limited in the amount of screen space they provide; such devices are meant to be carried on one's person, and therefore cannot be larger or heavier than allowed by portability restrictions. Scrolling and zooming are therefore even more important on hand-held devices than on larger computers, because so many more images, documents, etc., have to be viewed “one part at a time” on hand-held devices.
Some hand-held devices are now capable of detecting the amount that they are tilted with respect to vertical, and allow the user to control various features on the device by tilting it. Scrolling and cursor movement are already controllable in this way (e.g., Hitachi U.S. Pat. No. 5,602,566). However, the '566 patent merely uses tilt to allow users to choose which portions of an image appear on the hand-held display. Thus with traditional scrolling, the user can change the portion of image under view but cannot change the magnification or perspective of the image.
Other references use tilt for zooming, such as U.S. Pat. No. 6,201,554 to Ericsson, which allows zoom control by the user through movement of a hand-held device.
The motion-sensitive controls of the current invention provide a user interface that is more natural than current common controls because it allows a user to see more of an image at one time than traditional scrolling, viewing it in perspective so that he can relate new portions in view with portions that were in view already, and doing so with perspective views with which the user is already familiar. In addition, the motion-sensitive control simultaneously allows the user to change the magnification of the image using movements that are compatible with the tilt movements.
A variety of hardware based solutions for motion sensing exist today, all of which have drawbacks. The most common motion sensor used in handheld devices is the accelerometer. Typically the accelerometers measure the force of inertia caused by linear acceleration, combined with the gravitational acceleration at the point being measured.
Given a single axis force reading from an accelerometer, it is impossible to differentiate between gravitation and inertia. This separation problem is one of the weaknesses of an accelerometer-only solution. Many handheld devices are now shipping with three-axis MEMS accelerometers, which partially address this issue. A three axis accelerometer that is not moving will measure a total of about 1 G of force across its three axes. By measuring and comparing force across these three axes, we can make certain inferences about the orientation and position of the device, but only if we make certain assumptions. If we assume that the device can rotate freely but not accelerate, we can approximate its orientation with respect to a horizontal plane. If we assume that the device can accelerate but not rotate, we can approximate its linear acceleration, and therefore its 3D spatial position. If we assume that the device can rotate freely, and can only accelerate vertically, we can approximate both its orientation and its linear acceleration along the gravity vector. In practice, however, handheld devices typically rotate freely around all three axes, and move freely along all three axes, making it impossible to accurately model both the real world position and orientation of the device.
In addition to the gravitational/inertial separation problem, accelerometers suffer from an inability to detect rotation around the force vector. So, for example, a motion application that depended on measuring rotation of a stationary device around the device's Y axis would work quite well when the device is horizontal, would become less accurate as the angle between the Y axis and the horizontal plane increases, and would become unpredictable as the Y axis becomes aligned vertically with the gravity vector.
Because of these drawbacks, most devices that include accelerometers must make assumptions about how they'll be used, and must accept or work around the flaws inherent to this technology.
Future devices will likely combine both gyroscopes and accelerometers to form Inertial Measurement Units (IMU), which allow measurement of both rotation and linear acceleration. Gyroscopes can measure changes in orientation, but not position. This can be used to subtract the gravitational component from the accelerometer measurement, leaving only inertial force due to linear acceleration. Today, however, gyroscopes are too expensive to include in most consumer handheld devices such as mobile phones.
Many mobile devices lack motion sensing hardware, but include digital cameras that can take motion video, typically at a rate of 15 to 30 frames per second. On some devices, these cameras have been used as a sort of crude motion sensor. By comparing the overlap between consecutive frames of video, it is possible to approximate the change in orientation, assuming a relatively fixed device position and a stationary background with sufficiently high contrast. This optical solution was popular when accelerometers were more expensive, since it worked on existing devices without requiring new hardware, but ongoing price reductions in MEMS accelerometers, combined with the optical solutions high power consumption, low accuracy, and environmental constraints are increasingly leading today's manufacturers toward accelerometer-based solutions.
The user interface methods described in the current invention can be implemented, with certain limitations, using a pure accelerometer, or pure optical based sensor solution. The accuracy and responsiveness of these user interface methods could be improved by including a gyroscope, but this option is too expensive, as discussed above. Therefore, the current invention proposes the novel approach of combining accelerometer measurements with optical measurements of relative orientation to overcome the weaknesses of each individual technology, thereby improving the performance of the user interface methods described herein.